The primary goal of existing scene parsing techniques is to associate a class/label such as “sky,” “tree,” “dog,” etc. with each pixel in a still image. Labels, which may also be referred to as classes herein, are useful in assigning keywords to images, categorizing images, organizing images, and a variety of other purposes. The ability to automatically parse image scenes to assign labels to particular portions of an image can greatly reduce user effort in image editing and organization. Existing techniques for automatic scene parsing often fail to accurately classify or label interesting objects for various reasons. For example, rare objects, such as boats and people, are often not identified accurately.